outgroup in ancestral sequence reconstruction

[ga01]

Hi everyone,
I would like to ask for some advice on the use of outgroups in ASR: is there a way of instructing IQtree to ignore the outgroup in ASR?
For example, if I have a tree (A,B,C), where C was specified as an outgoup (with the -o flag):

|----- A

|--| (Node1)
| |-----B
|
|-------C

is there a way of telling IQtree to use only A and B in the reconstruction of the ancestral state of Node1? My understanding is that even if I specify C as an outgroup, it uses C in the reconstruction of the sequence of Node1, and one would need an additional node D to be able to avoid it:

    |----- A
|--| (Node2)
|   |-----B

|--| (Node1 - root)
| |-------C
|
|----------D

Thanks,
G.

[soupvector]

If, at a given position, A and B differ, the outgroup C would allow inference of the branch on which state change occurs. Would you prefer that all such positions would be left as multistate characters in the ASR?

[ga01]

You are right that in the case of two sequences one has to have an outgroup, so my example was too simple, but in the case of many sequences this is not necessarily the case, and one can make inferences about the root without knowing the state of the outgroup.

My dilemma is that if I have a large tree, with hundreds of sequences, but my outgroup is only a single sequence, than this sequence may have a disproportionally large effect on the reconstruction, and may introduce a significant bias. So maybe it is best not to use it at all (aside from specifying the root, i.e. the topology of the tree), and I wonder whether IQtree can do this.

[soupvector]

I'm also interested in replies to your questions - especially those that are informed by knowledge of the method for ancestral state reconstruction in IQTree2 - if the outgroup is distant and particularly if it's composed of multiple sequences (adding state ambiguity to the outgroup), it seems to me that the outgroup need not be ignored but rather carry less weight.

[bconisko-cyber]

I have 1st run iqtree2 on an alignment without an outgroup without requesting asr. Then opened the output treefile in FigTree, rooted it, and exported as output_rooted.nwk
Next I rerun iqtree2 using the original alignment and the rooted tree:
iqtree2 -s original.phy -te output_rooted.nwk -m LG+R4 -asr -T AUTO --prefix xyz_
It runs with no errors. But what nodes correspond to what I find in the state file? (The output_rooted.nwk tree has no nodes assigned, and the output treefile from the run described above is NOT rooted, and has no node for the earliest ancestor.)

Help!