Both files out1.txt and out2.txt contain the same numbers, which I believe are the averages on all the atoms. How do I get the average of the atoms only in the group “test”?
Fix ave time works on global scalars and vectors, i.e. it takes the whatever input you feed it and averages it. Your input are variables defined using the termodynamic output, so it takes what is computed there (which is typically for all atoms). You cannot partition these numbers into subsets after the fact.
You first have to compute these properties for only that group and the average the output of that computation.
I see the following quoted text in the fix ave/time do page, which takes precedence over the generic fix documentation since it is specific for this fix. Also you are only looking at the examples and summaries; you have to read the entire documentation.
Description
Use one or more global values as inputs every few time steps, and average them over longer timescales. The resulting averages can be used by other output commands such as thermo_style custom, and can also be written to a file. Note that if no time averaging is done, this command can be used as a convenient way to simply output one or more global values to a file.
The group specified with this command is ignored. However, note that specified values may represent calculations performed by computes and fixes which store their own “group” definitions.
Feel free to have a pass at the sources for the manual, identify all cases where this is the case, make the necessary changes and submit them as a pull request. The LAMMPS developers will certainly appreciate such an effort more than your suggestion. This is the beauty of Open Source software: if something bothers you, you can take action right away and help to make the bother go away.