We recently described the cloning of a fifth member of the 5-hydroxytryptamine (5-HT)1 (serotonin1) receptor class that inhibits adenylyl cyclase, namely the human 5-HT1F receptor (Adham et al. 1993a). In the present study we have examined in greater detail the functional coupling of the 5-HT1F receptor in two different cell lines, NIH-3T3 and LM(tk-) fibroblasts (receptor densities of 1.7 and 4.4 pmol/mg protein, respectively). The maximal inhibitory response elicited by 5-HT was significantly greater in NIH-3T3 as compared to LM(tk-) cells, whereas the EC50 values were comparable. To investigate the relationship between receptor occupancy and inhibition of cAMP accumulation mediated by 5-HT1F receptors in NIH-3T3 cells (and hence the degree of receptor reserve), we used the irreversible receptor antagonist N-ethoxycarbonyl-2-ethoxy-1,2-dihydroquinoline (EEDQ). The half-maximal response required only about 10% receptor occupancy, consistent with a receptor reserve of 90% (88 +/- 2.1%, n = 4) for 5-HT-induced inhibition of FSCA. Despite the presence of such a high degree of receptor reserve, a range of intrinsic activities was displayed by structurally diverse classes of compounds. For example, sumatriptan and lysergol were as efficacious as 5-HT itself and thus acted as full agonists, whereas metergoline and 1-NP behaved as partial agonists and as shown previously (Adham et al. 1993a), methiothepin was a silent antagonist (Kb = 438 nM). We have also investigated activation of additional signal transduction pathways by the 5-HT1F receptor and found that the responses differ in the two cell lines with respect to stimulation of phospholipase C. For example, in NIH-3T3 cells no elevation of inositol phosphates (IP) of [Ca2+]i was observed even at very high agonist concentrations (100 microM). In contrast, in LM(tk-) cells concentrations of 5-HT as low as 10 nM induced stimulation of IP and a rapid increase of [Ca2+]i. The 5-HT1F receptor failed to alter arachidonic acid release in either cell line. The maximal increase in IP accumulation in LM(tk-) cells was modest, averaging about 100% above basal. The increases of IP and [Ca2+]i required 5-HT concentrations less than one order of magnitude greater than those inhibiting FSCA (EC50 = 17, 55 and 8 nM, respectively), and both responses were blocked by 100 microM methiothepin. All three responses (cAMP, IP, and [Ca2+]i) were sensitive to pertussis toxin pre-treatment, suggesting the involvement of Gi/Go protein(s) in these signal transduction pathways.(ABSTRACT TRUNCATED AT 400 WORDS)