The problem was - users don't want to chose options in between FBA login and NTLM login.
First: User who is FBA user should see only FBA Login dialog and AD users want to login transparently, then they on LAN and already authenticated.
Second: Then you login using standard SharePoint FBA login page, it shows you red alert about - user name and password will be send in clear text over the connection.
- Instead creating AAM we need to extend web application:
- For example - create a web application with CBA (Claim Based Authentication) with NTLM and FBA providers. - name it MYTEST on port 80 - http://mytest . It will be default zone.
- Give at least one FBA user full access to the site. Extend the web application as http://mytest.mydomain.com on port 80 with only FBA providers. It will be Extranet zone.
- For configuring SSL access (to remove red alert from FBA login page):
- Go to AAM collection for the web application and edit http://mytest.mydomain.com to https://mytest.mydomain.com
- Add binding in IIS to https (you need to link Public SSL certificate to http://mytest.mydomain.com).
- For the certificate - Wild card certificate can be used.