Can I have an attribute-based and a class-based property drawer on the same field?

I have a custom class:

 [Serializable]
 public class CustomType { ... }

And a custom drawer assigned to it:

 [CustomPropertyDrawer(typeof(CustomType))]
 public class CustomPropDrawer : PropertyDrawer { ... }

But then I have a custom attribute as well:

 public class CustomAttributeAttribute : PropertyAttribute { ... }

And a drawer for that:

 [CustomPropertyDrawer(typeof(CustomAttributeAttribute))]
 public class CustomAttributeDrawer : PropertyDrawer {
     public override void OnGUI(Rect position, SerializedProperty property, GUIContent label) {
         ...
         EditorGUI.PropertyField(position, property, label, true);
         ...
     }
 }

When I make an object with a field that contains both the custom type and the custom attribute, all hell breaks loose:

 public class ClassWithCustomType : MonoBehaviour {
     [CustomAttribute]
     public CustomType value;
 }

More specifically, the custom attribute drawer renders correctly, but the custom type drawer is not called, and the default struct renderer is used to ultimately draw my custom type instead. What am I doing wrong here? Can I even do what I desire to do?