Can AI Classify Stars?

This project follows the Engineering Design Process. Confirm with your teacher if this is acceptable for your project, and review the steps before you begin.

Overview

In this project, you will preprocess the data by eliminating data or columns deemed unnecessary by statistical analysis. You will also adjust parameters of a boosted tree model to achieve the highest accuracy in classifying celestial objects into categories (e.g., STAR, GALAXY, and QSO).

Setting Up the Google Colab Environment

1. You will need a Google account. If you do not have one, make one when prompted.
2. Download the stellar_classification.ipynb file from Science Buddies. This is the code you will need to process your data.
3. Download the stellar_data.csv file from Science Buddies.
   1. Open the dataset to familiarize yourself with its structure by going to your Downloads and double-clicking on the file.
   2. You can learn more about each column by clicking this link here.
4. Within your Google Drive, click on ‘MyDrive,’ then create a new folder and rename it “stellar_classification.” Inside the folder, upload both the stellar_classification.ipynb file and the stellar_data.csv file.
5. Double-click on the stellar_classification.ipynb file. This should automatically open in Google Colab.
   1. Read the Troubleshooting Tips and How to Use This Notebook sections. Follow the instructions you find in that section.
   2. Run the block under Importing Libraries to ensure you have access to all the functions we will use for this project.

1. Loading the Data into a Pandas DataFrame

1. (Code Block 1A) Run this code block to create a DataFrame, which is like a table that will be used to load and manipulate the data in the notebook. You will see the data from your .csv file populate in a table below the code block. 

2. Preprocessing

2.1 Label Encoding

When working with machine learning algorithms, it is important to convert categorical variables into numbers since algorithms understand numbers better than words. Categorical variables represent qualities like gender or group, and cannot be used because they lack a numerical form that algorithms can work with. Label encoding is a way to convert categorical variables into positive whole numbers, assigning each category a unique integer value (0, 1, 2, etc.).

1. (Code Block 2A) This code block converts the labels in the “class” column (e.g., “GALAXY”, “STAR”, and “QSO”) into numbers (0, 1, 2) using a mapping dictionary. It ensures the data is ready for machine learning. Run this code block.

2.2 Detecting Outliers

Outliers are data points that differ significantly from the majority of values in a dataset. Detecting and removing them is important because they can skew results, reduce model accuracy, and hide true patterns in the data. Outliers often result from errors, anomalies, or rare events and can mislead analysis if not addressed. Removing them helps make data cleaner and improves reliability, but it is important to check if they provide valuable insights before removing them. 

In our case, outliers in celestial object data can arise from both natural and observational causes. Natural causes include unusual objects like rapidly rotating stars, active galaxies, or rare events such as supernovae and gamma-ray bursts. Observational causes include measurement errors, contamination from nearby objects, blending of light from multiple sources in crowded regions, and issues with data processing or telescope equipment. Sometimes, human mistakes, such as mismatching objects in catalogs or using incorrect models, can also create outliers. These need to be carefully analyzed to determine whether they represent real phenomena or just errors.

1. (Code Block 2B) This code block uses the Local Outlier Factor (LOF) algorithm to detect outliers in a dataset. It creates a LOF model, fits it to the dataset, and predicts whether each data point is normal (1) or an outlier (-1). LOF identifies outliers by comparing the density of each point to its neighbors, making it useful for spotting anomalies in data. Click on this link to learn more about the LOF algorithm. 

2. (Code Block 2C) This code block detects outliers using scores from the LOF model. It stores these scores, sets a current threshold (-1.5), and identifies points with scores below the threshold as outliers. Finally, it will count and print the total number of outliers.

3. (Code Block 2D) This code block removes rows identified as outliers from the LOF model. This ensures that the DataFrame is cleaned by removing outliers, leaving only inliers for further analysis or modeling. Run this code block.

2.3 Feature Selection

1. (Code Block 2E) This code block generates a heatmap to help easily identify relationships between features in the dataset, such as strong positive correlations (closer to 1) or negative correlations (closer to -1).
   1. Which features are most correlated with one another? Which ones have the weakest correlation?
2. (Code Block 2F) This code block calculates how strongly each numerical feature in the DataFrame is related to the ‘class’ column using correlations. It then sorts these correlations, showing which features are most positively or negatively related to ‘class.’ This helps in selecting important features and understanding the data.
3. (Code Block 2G) This code block removes specific columns from the DataFrame. Inside the list under the #TODO comment, add the column names of the columns you want to drop. Here are some things to consider:
   1. Features with very low correlation (close to 0) have minimal impact on predicting class.
   2. Retain columns with higher correlations (close to 1), as they are more strongly related to the target (class).
   3. Consider keeping features with negative correlations if the absolute value of the correlation is significant. A negative correlation indicates an inverse relationship with the target variable (class), which can still be valuable for prediction.
   4. Drop columns with missing values or NaN correlation unless you can impute or determine their importance through other means.

2.4 Handling Imbalanced Data

Training on a balanced dataset is important because it ensures that the model learns equally from all classes and doesn't become biased toward the majority class. In an imbalanced dataset, the model may prioritize accuracy by focusing on the dominant class while ignoring the minority classes, leading to poor performance for underrepresented categories. A balanced dataset improves the model’s ability to generalize and make fair predictions across all classes. There are many ways to combat imbalanced data, and in this project, we will be using an oversampling method called SMOTE, which will be explained in more detail below.

1. (Code Block 2H) This code block creates a count plot to show how many entries belong to each category in the ‘class’ column. Each bar represents a unique class, and its height shows the count of entries for that class. The plot helps you see if the dataset is balanced or has more entries in some categories than others.
2. (Code Block 2I) This code block splits the data into input features (X) and the target variable (y). It removes the ‘class’ column to create X, which contains all the other data, and selects the ‘class’ column for y, which is the target for prediction.
3. (Code Block 2J) This code block applies the SMOTE (Synthetic Minority Over-sampling Technique) algorithm to balance an imbalanced dataset by generating synthetic samples for the minority class. Click on this link to learn more about SMOTE.
4. (Code Block 2K) This code block creates a count block to show the distribution of target variable y (class labels) after resampling using the SMOTE technique. This plot helps confirm if the dataset is balanced after resampling.
5. (Code Block 2L) This code block scales the input features (x) to standardize them, so they have a mean of 0 and a standard deviation of 1. This helps machine learning models work better and train faster by putting all features on the same scale. Click on this link to learn more about feature scaling.

3. Split to Train and Test

We will split our dataset into train and test to avoid overfitting and ensure the model is tested on data it has never seen before. Click on this link to learn more about splitting our data into train and test.

1. (Code Block 3A) We have provided the code to split the dataset into training and testing parts. Run this code block.
2. (Code Block 3B) This code block prints out the sizes of X_train, y_train, X_test, and y_test. If you see the X_train size as (121668, 9), that means there are 121669 samples (celestial objects) in the training data, each with 9 features (e.g., redshift, plate, MJD, etc.).

4. Training the Model

1. (Code Block 4A) We have provided the code to create and train a boosted tree classifier. The accuracy of the model will be printed below. Run this code block.

5. Evaluating the Model

1. (Code Block 5A) This code block classifies data using a confusion matrix, and displays the heatmap of correct and incorrect predictions for each class (GALAXY, STAR, QSO). This helps identify the model’s accuracy and errors. Click on this link to learn more about how to read a confusion matrix.
   1. Are there any classes the model struggles to classify correctly? Why might this be?
2. (Code Block 5B) This code block displays a classification report to evaluate the performance of the model on the test dataset. The report includes key metrics for each class:
   1. Precision: Measures how many of the predicted positive instances are actually correct.
   2. Recall: Measures how many of the actual positive instances were correctly predicted.
   3. F1-score: The harmonic mean of precision and recall, balancing the two metrics.
   4. Support: The number of true instances for each class in the test set.
3. (Code Block 5C) This code block evaluates and visualizes the ROC curves to show the balance between true positive and false positive rates, and the area under the curve (AUC) scores indicate the model’s performance for each class. Higher AUC scores mean better classification. Click on this link to learn more about ROC and AUC.
4. (Code Block 5D) This code block shows how well the model predicts each class (GALAXY, STAR, QSO) using a bar chart. The chart compares actual labels to predictions and highlights mistakes.
   1. Which class has the most prediction errors? Why do you think this is the case?
   2. Are there any classes the model predicts accurately most of the time? What might contribute to this accuracy?

6. Refine Your Model

Now, it is time to experiment and improve the model by adjusting various settings, such as the threshold for outliers, learning rate, which features to include or exclude, maximum depth of the trees, and the number of trees in the ensemble. Remember that accuracy can be found in Code Block 4A and that all code should be run again when adjusting parameters.

In a notebook or spreadsheet, create a table like the one below to keep track of changes and its affect on the model's accuracy:

1. Adjust the threshold for outliers (Code Block 2C)
   1. Try adjusting the threshold value to see how that affects the number of outliers. Remember that LOF assigns scores close to -1 for inliers. Scores significantly lower than -1 (e.g., below -1.5 and -2) often indicate outliers.
   2. A common starting point is -1.5, but it is important to adjust it based on the dataset and the desired level of sensitivity.
2. Adjust the learning rate (Code Block 4A)
   1. The learning rate is a hyperparameter, which is a value set before training that controls the behavior of the training process and is not learned from the data. The learning rate determines the size of the steps the algorithm takes when adjusting the model’s parameters (weights that do change with training). Click on this link for a quick explanation of learning rate.
      1. A learning rate that is too high might overshoot the optimal solution, fail to converge, or exhibit erratic behavior.
      2. A learning rate that is too low may take a long time to converge or get stuck in a local minimum.
3. Final Step: Iterate and Track Your Results
   1. Throughout your experimentation, keep a detailed log of your adjustments and their corresponding results in the data table shown above.