@JooHyukKim Ok, this might be the way to do it, but we should actually change base (de)serializers from jackson-databind -- ReferenceTypeSerializer and ReferenceTypeDeserializer. That way AtomicReference would be covered, as would Scala Option. Possibly no change needed here, except for testing.
That would also simplify merging as master/3.0 has Optional directly supported by jackson-databind itself (2 out of 3 modules from here merged).

I also wonder if JSON serialization should (or should not) be changed for 3.0.
With 2.x we will basically omit Optional level altogether (both regular and polymorphic cases). But I think this was a mistake I made: I think better way would have been to add level for polymorphic case to have:

{
  "java.util.Optional": {
      "com.fasterxml.jackson.datatype.jdk8.PolymorphicOptionalTest$Foo":{"bar":1}
  }
}

instead of what 2.x uses and expects:

{"com.fasterxml.jackson.datatype.jdk8.PolymorphicOptionalTest$Foo":{"bar":1}}

Obviously such change cannot be done for 2.x, but could be done for 3.x.
But even if change was made, we should probably add a MapperFeature for allowing enabling of old (2.x) behavior.

First things first however: this fix should be applied to jackson-databind base handlers.

but we should actually change base (de)serializers from jackson-databind -- ReferenceTypeSerializer and ReferenceTypeDeserializer

Yeah this makes sense. Let's work through databind module instead.

I think better way would have been to add level for polymorphic case to have:

What benefit are we expecting out of the described serialization form?

@cowtowncoder Filed fix PR in databind module FasterXML/jackson-databind#4838 the PR contains how to test integrated etc...

I think better way would have been to add level for polymorphic case to have:

What benefit are we expecting out of the described serialization form?

There may be multiple issues but the problem with Optional serialization wrt type info is that there are two things that could be polymorphic:

1. Contents of Optional. I think this is more about that and is not problematic (as long as Optional itself is known as type from enclosing class
2. Type of Optional itself, f.ex when nominal type is java.lang.Object and default typing is enabled. I think this would not work.

But I must admit that the interaction gets pretty complicated and fundamentally it might be that it is not possibly to only add new level for polymorphic case.
In fact original "unwrapped" approach to Optional serialization might be a bad decision form my part. So, for example, for Optional<Integer>(5) is now serialized simply as

5

and Optional<Integer>(null) (or rather Optional.empty()) as

null

but it 'd have been simpler, in the end if those instead were serialized as

[ 5 ]

and

[ ]

because then we wouldn't need convert null into Optional.empty() (plus actually be able to distinguish null and Optional.empty() when deserializing).

But it may be bit late to change this even for 3.0. And there hasn't been too many reports for problems (there are some, wrt nested Optionals but not a ton.

So let's focus on immediate issue and forget about what I was proposing for now. :)

Closing due to reasons I mentioned in comment