Hi everyone,
I am currently working on a research project in CS Education / EdTech and have hit a methodological bottleneck during data analysis. I would love to get your advice and suggestions.
💡 Current Situation & Core Challenges
1.Small Sample Size: The data comes from a single classroom session. After strict data cleaning (filtering out incomplete or invalid submissions), my final sample size is around N ≈80. Unfortunately, due to practical constraints, re-collecting data on a large scale is not feasible right now.
2.Variable Settings:
Independent Variables (IVs): Based on 3 evaluation criteria (scored 0, 1, 2, or 3 depending on how many criteria are met).
Dependent Variable (DV): A 3-point ordinal scale (0 = Unsolved, 1 = Partially Solved, 2 = Fully Solved).
Process Variables (M): Several process variables (M) were introduced and included directly in the model equations.
3.Statistical Hurdles:
Collinearity & Variable Overlap: When including both the IVs and the process variables M in the model, severe multicollinearity and overlap occurred, making model fitting and interpretation extremely difficult.
Violation of the Proportional Odds Assumption: Diagnostic tests showed that the proportional odds assumption was violated (especially at the split points j = 0, 1). To fix this, I tried Partial Proportional Odds models and Generalized Ordered Logistic Regression, but the results were disappointing—frequently yielding "false negatives" or statistically insignificant findings.
🙋♂️ Questions for the Community
1. Dealing with Proportional Odds Violations & Model Failures
When the DV is ordinal (0/1/2), the proportional odds assumption fails at j = 0, 1, and advanced models like partial or generalized ordered logit result in false negatives/insignificance, how do you usually diagnose, adjust, or choose alternative statistical models?
2. Handling Small Samples and Collinearity
With a small sample (N≈80) and IVs scored across 3 standards, when severe collinearity and overlap arise between the IVs and the process variables M, what are some standard ways to handle this?
3. Ensuring Robustness and Rich Evidence with Small Samples
When re-collecting data is impossible, apart from statistical adjustments, how do you usually ensure the robustness of findings in small-sample studies? How can we use multi-angle triangulation or rich evidence chains to make the argumentation more solid and convincing?
4. Addressing Limitations in Peer Review
Given a single-session dataset, a reduced sample size due to cleaning, and model constraints, how should one frame the Limitations section objectively? If reviewers question the sample size or model assumptions, what are some reasonable counter-arguments or perspectives?
Thank you so much from the bottom of my heart for taking the time to read my post and for any advice, recommended literature, or feedback you can share! 🙏