"Maggie has asked Julian to write her character out," the U.K.'s Daily Mail is reporting. "She is filming until August and then wants to leave, going out on a high. She thinks that three series is enough and she wants to get back to the stage and big screen."
Also read: 'Downton Abbey' Season 3 Death: Who Is It?
Smith's Dowager Countess, a master of the potent one-line put-down, is one of the show's pivotal characters. The Daily Mail's source says she is likely to be struck down by a fatal illness at the end of the third season, which ITV is currently filming in the U.K.
Smith, 77, won a Best Actress Oscar for playing a troubled teacher in the 1969 film "The Prime Of Miss Jean Brodie." She may be best known to today's audiences for her roles in the Harry Potter movies
In November, she'll be seen in the BBC Films comedy "Quartet."