Heart disease is the leading cause of death in the United States, the UK, and worldwide. To improve the prediction and prevention of heart disease, risk factor identification is the main step in diagnosing and preventing it. Many studies have attempted to detect risk factors for heart disease, but none have identified all risk factors. The National Center for Informatics for Integrating Biology and Beyond (i2b2) proposed a clinical natural language processing (NLP) challenge in 2014, with a track (track2) focused on detecting risk factors for heart disease risk factors in clinical notes over time. This paper presents an improved approach to the 2014 i2b2 challenge by identifying tags and attributes relevant to disease diagnosis, risk factors, and medications by providing advanced techniques of using stacked word embeddings. The proposed model achieved an F1 score of 93.66% by using BERT and character embeddings (CHARACTER-BERT Embedding) stacking.