Blending families can be complicated — and The Brady Bunch’s Christopher Knight isn’t sure his beloved series always got it right.

The actor, who played middle son Peter Brady, spoke about the titular family’s sometimes odd dynamics on the April 21 episode of The Real Brady Bros podcast, which he co-hosts with on-screen big brother Barry Williams, a.k.a. Greg Brady. In the episode, in which they discussed the first episode of The Brady Bunch, the hosts were joined by Susan Olsen, who played Cindy Brady, and Mike Lookinland, who played Bobby Brady, the two youngest siblings.

The episode featured the wedding between Florence Henderson’s Carol and Robert Reed’s Mike. The cast members noted that Carol and her three daughters — Maureen McCormick’s Marcia, Eve Plumb’s Jan and Olsen’s Cindy — all changed their last names to Brady, even though viewers never saw Mike actually adopt them in an episode. 

Williams, 70, also said, “We don’t ever refer to ourselves as stepbrother, stepsister, or stepmom.” He “credited” the show’s creator, Sherwood Schwartz, with that decision. 

Olsen, 63, said that the writers eventually “forgot” they weren’t siblings by blood. “They had Jan say to Peter, ‘You’re my brother, blood is thicker than water,’ ” she remembered. “And Eve and Chris went to the producers and said, ‘We’re not related.’ ” But the producers said “nobody” remembered that, she added. 

“I took offense to it,” Knight, 67, said. “I thought that was the part of the show that really failed us as kids. And I was just 12 years old, thinking at that time, ‘They’re not really my sisters.’ And I know growing up, I would have said, ‘You’re not really my sister.’ ” As a child actor, he found it confusing that the Brady kids never used this tactic when they had fights.

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.  

But now as an adult, he said, “Looking back on the show and its success, which is undoubtable, I recognize that the reason in part for that success is because we didn’t do that. It was all about getting along.” If the Brady boys had a photo of their birth mom up (which did happen in the pilot episode, only to never be seen again), it would be “a potential wedge issue,” he added.

“So it was completely put aside,” Knight said.

Later in the episode, Williams noted that the idealistic blending of families was “Sherwood’s whole idea” for the series. He had read a news article about “the number of families in the ’60s that the parents were getting divorced,” leading the kids to be home alone as “latchkey kids.” The costars also noted that in many episodes of the show, “Carol’s kids” and “Mike’s kids” got into fights as factions.