The COVID-19 pandemic continues impacting all segments of the global population, causing many problems, from health and wellbeing issues to definite irretrievable damage to the society. Despite the need for a quick and accurate response for early risk stratification and diagnosis, rare and novel diseases, e.g., COVID-19, are very difficult to diagnose. Although deep learning diagnostic algorithms have shown promising results in a wide range of tasks, they require a massive amount of labelled data for training. However, due to the nature of novel diseases, availability of such huge amount of well annotated data poses a great challenge to the learning algorithms. Motivated by this, in this work, we present an open-source deep meta learning solution based on siamese convolutional networks, called COVID-Net FewSE, that is able to detect COVID-19 positive cases from a limited number of X-ray images. Trained on the COVIDx-CXR dataset, the model achieves 0.9 recall and accuracy of 0.997 in detecting COVID-19 cases from X-ray images, when only 50 training samples are available. Our experimental results confirm that the proposed model outperforms conventional machine/deep learning classifiers in COVID-19 detection when limited samples are available. The model and all the scripts are made available to the public to enable reproducibility and encourage further innovation in the field.