In the New England town of Stoningham, widow Cary Scott is disappointed when her friend, Sara Warren, cancels a lunch date, and so invites her landscaper Ron Kirby to share the meal. Immediately, Cary is drawn to Ron's strength and calm, but his youth and blue-collar social status make a romance unthinkable to her. That night, Cary's children, budding executive Ned and co-ed Kay, come home and grant their approval to Cary's date with the sole local bachelor, staid hypochondriac Harvey. Cary and Harvey go to the country club, where a new neighbor, Tom Allenby, is already being targeted by a young blonde woman. After neighborhood gossip Mona Plash criticizes Cary's red dress as inappropriate, the married Howard Hoffer makes a pass at Cary, who deflects it. Harvey takes her home and there proposes to her, but Cary, who yearns for some of the passion she felt with her husband, demurs. Weeks later, Ron returns to prune the trees, and Cary is surprised at the disappointment she feels after he announces he is quitting in order to run his tree farm. When he asks her over to see his trees, she reluctantly agrees, only to be charmed by his rustic greenhouse cabin and down-to-earth manner. As she explores the abandoned mill next door, a bird frightens her and she falls into Ron's arms. Cary then turns to leave, but Ron stops her, and they share a passionate kiss. Weeks later, autumn progresses, and Cary, horrified by Sara's advice to buy a television set to keep her company, accepts Ron's invitation to a dinner party. It is held at the home of his friends, Alida and Mick Anderson, former suburbanites who, at Ron's urging, have turned to living on the land for fulfillment. Alida explains that they live by the words of Henry David Thoreau: "If a man does not keep pace with his companions, perhaps it is because he hears a different drummer. Let him step to the music he hears, however measured or far away." As the party begins and she dances with Ron, a joyful Cary realizes how much she admires Ron's lifestyle and strength of mind, and feels a twinge of jealousy when she sees Alida's pretty niece, Mary Ann, flirting with him. By winter, Ron and Cary are spending all their time together, and he surprises her by showing her how much of the mill he has remodeled. When he tells her that he has built the house for them to share, however, she insists that the union would be impossible, because her friends and children would not accept him. She begins to leave, but breaks down crying, and soon after they declare their mutual love. Within days, Mona spies them together and spreads a rumor that they began their relationship before Cary's husband died. The faithful Sara suggests that Cary bring Ron to a party that weekend so their friends can meet him, but at the party, the local couples disdain Ron as "the gardener" and snub him. After Howard declares Cary a tease, Ron slugs him, and the couple quickly leave. At home, Cary tells Ned and Kay that she is going to marry Ron, and although they were amenable to her relationship with Harvey, they are horrified to think she might marry "beneath" her and sell the family home. After Kay cries that her life has been ruined by the gossip and Ned threatens never to return home, Cary tells Ron they must wait to be married. He demands that she choose between her love for him and her need for social acceptance, and even though she is devastated, Cary leaves Ron. Weeks later, her friends and family have welcomed Cary back into their fold, but she remains despondent and suffers headaches. Soon, the kids are too busy to visit, and a lonely Cary is crushed when she sees Ron and Mary Ann together. At Christmas, Kay shows off her engagement ring and Ned announces that he is moving to Paris and wants to sell the family house. When Cary sees their gift, a TV set, she breaks down, realizing that her rejection of Ron was pointless, and her future holds only loneliness and boredom. The next day, she visits Dr. Dan Hennessy, who opines that her headaches are caused by depression, and that she should marry Ron. Although she goes to Ron's, she hesitates at the door and returns to her car. Ron, who has been hunting, spots her from atop a hill and, in his rush to stop her from leaving, falls off a cliff and suffers a concussion. That night, Alida informs Cary that Ron is unconscious, and they race to his cabin, where Cary admires the beautiful home Ron has built and anguishes over why it has taken her so long to discover her true values. When Ron finally wakes the next morning, he is delighted to see Cary, who assures him that she has finally come home.