Introduction: Ultrasound examination is a medical diagnostic test broadly used, due to its properties of being a non-invasive method and its low operational costs. Commonly, Radiologists perform ultrasound examinations, as a part of their medical specialty. However, ultrasound examinations could be applied by other specialties, as a first screening test.
For that, a Training program for General Practitioners (GPs) leaded by Radiologists was designed, and the inter-rate agreement and reliability among GPs and Radiologists was analyzed in order to implement ultrasound examination performed by GPs at Primary Care Centers (PCC). Also, the level of patient' confidence on getting an ultrasound examination done by GPs, instead Radiologist, was measured by a satisfaction survey.
Methods: 60-hours course of theoretical and practical training was done, followed by 24-weeks practical sessions at the Radiology Service. Thirteen GPs were enrolled on the Training program. On practical sessions, both GPs and Radiologists assessed each patient and recorded the data on a newly built electronic form. Kappa index was calculated including the overall inter-rater agreement among GPs and Radiologists examinations.
After Training program completion, GPs used ultrasound examination in their Primary Care Centers and patient's satisfaction survey was done.
Results: The total number of patients assessed by both professionnels was 1,928: 1,377 patients had abdominal ultrasound scan, 345 patients had urological ultra Sound scan, and 206 patients had thyroid ultrasound scan. A high degree of agreement among raters was observed (κ ≥ 0.8) in most of the double-checked organs. The patients overall assessment rate was very satisfied: 4.8 out of 5.

Conclusions: This study provides insights into ultrasound examination used by GPs in PCC, after an appropriate training. That could be used as fundamental data in improving health care protocols in the near future.

Keywords: Ultrasound, General Practitioner, Radiologist, Primary Care, Training, Guidelines, Inter-Rater Agreement, Reliability

Ultrasound examination is a rapid, non-invasive method, and radiation free investigation with a wide variety of clinical applications. It is a medical test used for diagnosis, due to its properties of harmlessness, good tolerance, dynamics and low cost [1]. Traditionally, ultrasound examinations have been a part of the Radiology specialty.
However, in recent years, other medical specialties have adapted the use of ultrasound examinations for their patients. This way of conducting ultrasound examinations is known as a point-of-care ultrasound examination [2]. Since long time ago, a few publications looked into the possibilities of utilizing ultrasound examinations in PCC [3]. Interestingly, there are a few studies where General Practitioners (GPs) of Primary Care Centers (PCC) had been trained and they use point-of-care ultrasound examinations daily [4-10].

In Spain, Radiologists have been the medical professionals who used to perform requested ultrasound examinations in Primary Care Centers (PCC). On 2012, the Spanish Ministry of Health recommended to incorporate ultrasound examination in PCC [11]. Gradually, PCC has been equipped with ultrasound machines and training for GPs has been provided. However, a structured education and training on ultrasound technology is mandatory to keep skills at a sufficient level over time for GPs.
An ultrasound examination-training program for GPs was designed by the Primary Care Management in Lleida (DAP Lleida), and aligned within the current framework of the Catalan government Health plan (2011-2015) [12]. For that, the recommendations of the Royal College of Radiologists in London and the European Federation of Societies for Ultrasound in Medicine and Biology were reviewed and adapted [13,14], and a specific training program for GPs leaded by Radiologists was organized to educate them on handling of ultrasound machines and processing of the obtained data for clinical diagnosis at PCC.
Thirteen GPs were enrolled on the full Training program, where 1,928 patients were assessed. The aim of the current study was to evaluate the quality of the ultrasound examinations performed by GPs after completion of the training program, and analyze the inter-rater agreement and reliability within GPs and Radiologists in a limited range of ultrasound examinations. Also, the patients' satisfaction with the ultrasound examination in a PCC was measured by a survey after the technique implementation.
This is the first big study that validates the use of ultrasound examination by GPs. GPs and Radiologist showed a very high degree of agreement and reliability in the performance of the technique as a first level screening.

Ultrasound examinations were performed with an ultrasound device. The ultrasound equipment had a Digital Imaging and Communication in Medicine (DICOM) license that allowed GPs and Radiologists to save all the recorded pictures in the system, allowing them to be double check for training purposes. Based on the Ultrasound Training Competency Assessment Sheet model [14], an electronic form was generated, where both professionals could record the obtained data in each ultrasound examination. After patient examination, each professional had to classify it as 'normal' (or benign entities) or 'abnormal' (presence of pathological signs). The Radiologist diagnose was taken as a correct option in all the cases, because GPs were still under training. The designed electronic form contributed to the reliability of data collection and to the study's accuracy.
The total number of patients who went under ultrasound examination was 1,928: 1,377 patients had abdominal ultrasound scan (bile duct, gall bladder, liver, pancreas, aorta, kidneys, spleen, bladder), 345 patients had urological ultrasound scan (kidneys, bladder and prostrate in men), and 206 patients had thyroid ultrasound scan (Figure 1). Gynecological/obstetric ultrasound examinations, oncology-related patients and advance liver disease patients were excluded from Patients cohort.

Once the 13 GPs finalized the Training Program, they used ultrasound examination in their medical consultation offices at Primary Care Centers (CAP Balaguer, CAP Cervera, CAP Cappont). When an ultrasound examination was necessary, it was performed by a GP and afterwards patients were asked to fill out a short written survey on patient's satisfaction. The survey was anonymous and it was composed by six-questions. The patients could grade the medical service in the consultation scoring from 1 (lowest score, not satisfied) to 5 (highest score, fully satisfied): 1) Was the waiting time for the ultrasound examination adequate? ; 2) Rate the comfort experienced during the ultrasound scan; 3) Were you properly informed about ultrasound examination? ; 4) Rate the level of confidence that an ultrasound is done by GP; 5) Rate the care received from the GP; 6) Rate the overall assessment of the consultation. 170 patients gave feedback on the consultation about the potential inclusion of ultrasound examination performed by GPs at the PCC.

For statistical evaluation, Kappa statistic index values were calculated including the overall inter-rater agreement between the results from the GPs and Radiologists examinations. Measurement of the extent to which data collectors (raters) assign the same score to the same variable is called inter-rater reliability. Kappa value was calculated on each ultrasound examination performed by GPs and Radiologists. Kappa index can be interpreted as follows: 0.61≤ κ ≥0.80 as substantial/'good' degree of concordance, and 0.81≤ κ ≥1.00 as almost perfect agreement/'very good' degree of concordance. The risk ratio, sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) were studied concurrently. The Radiologists assessment was considered the correct one. All patient data was anonymous.
SPSS 15.0 software was used for data management, statistical analysis and plotting of the results. Statistical difference was considered significant at p-values ≤ 0.05.

After the attendance to the Ultrasound Training program, thirteen GPs had enrolled to a practical 24 weeks program at the Radiology Service of two public Hospitals (Hospital de Santa Maria -HSM; Hospital Universitari Arnau de Vilanova -HUAV). The total number of ultrasound examinations was 4,107: 3,289 examinations were performed at the HUAV and 829 examinations at the HSM. More precisely, GPs performed a total number of 2,037 examinations (71 were excluded) and Radiologist performed 1,938 examinations (61 were excluded). The double assessed patient cohort was established with a total of 1,928 ultrasound examinations. Ultrasound examinations were classified on 1,377 abdominal, 345 urological and 206 thyroid ultrasound examinations (Figure 1).

The degree of concordance among GPs and Radiologists ultrasound examinations was good in pancreas (κ=0.629), thyroids (κ=0.749), left kidney (κ=0.792), right kidney (κ=0.780), and bladder (κ=0.789). The concordance was very good in bile duct (κ=0.821), gall bladder (κ=0.824), liver (κ=0.825), and spleen (κ=0.876). And it reached an almost perfect agreement in the aorta examination (κ=0.949) (Table 1).
The sensitivity of GPs examinations (defined as the percentage of positive correct identified) was very high in bile duct (97.7%), spleen (98.8%) and aorta (99.5%); high in liver (93.2%), gall bladder (92.5%), pancreas (93.7%), right kidney (89.7%), left kidney (89.3%) and bladder (98.1%). However, the lowest sensitivity was at thyroids (70.2%).
Precision and accuracy of the Training program in Ultrasound examinations was measured by the Positive/Negative predictive values (PPV/NPV). PPV was very high in pancreas (99.8%), aorta (99.8%) and bile duct (99.5%). A bit lower percentage was showed by left kidney (95.5%) and thyroid (95.2%). So, the Training program was adequate and validated.

However, some divergence indicators were also analyzed to know the extent of agreement among GPs ad Radiologist reliability. For that, the percentage of patients scored as 'abnormal' by the Radiologist and 'normal' by the GP (RADa) was calculated and it was very low (range 0.1% - 2.8%), and the risk ratio was also very low in all the examined organs (Table 2).
Moreover, pancreas showed the lowest concordance index and 6% of GPs examinations where not confirmed by the Radiologist. Supplementary data is provided on divergences among GPs and Radiologist for each organ (see Supplementary data) (Tables 3-11).

170 patients filled out a survey where they answered anonymously six questions. The overall assessment rating was very high (4.8 +/- 0.5), and the rate of confidence for an ultrasound performed by GPs was also really high (4.8 +/- 0.5). The survey showed direct feedback on the Training program and it was a legitimate indicator for improving the services and strategic goals in healthcare organizations (Figure 2).

This study analyzed the inter-rater agreement and reliability among General Practitioners (GPs) and Radiologist ultrasound examinations after specialized training program for GPs to potentially establish ultrasound examinations perform by GPs at Primary Care Centers (PCC), as a screening technique in medical consultations. Both GPs and Radiologist assessed a total of 1,928 examinations, during the practical phase of the training program. Ultrasound examinations were classified on 1,377 abdominal, 345 urological and 206 thyroid ultrasound examinations. A high degree of agreement among raters was shown (κ ≥ 0.8) in most of the double-checked organs. The patients overall assessment rate was very satisfied (4.8 +/- 0.5) after the ultrasound examination was done by a trained GP in a PCC. The present research results can be utilized as reliable data to define effective and new protocols in Primary Care in the near future.
In the current study, the total number of double assessed patients (N= 1,928) was higher than in previous studies, where a much lower number of patients were double-checked by a GP and a Radiologist (N=120, [15]; N=114, [2]. This characteristic gave stronger statistical power and relevance to our current data (Figure 1).
Interestingly, a high sensitivity was observed (89%) in most of the double-checked organs (Table 1). The lowest concordance was found in pancreas (κ=0.629; NPV=50%) and thyroid gland (κ=0.749; Sen=70.2%). This could be due to a lack of specific training in those organs, where the physiological complexity is intrinsically high. However, pancreas inter-rater agreement in the current study was higher than in previous ones (κ=0.38; [15]). On the other hand, on abdominal/urological areas a similar concordance levels were observed when the ultrasound examination was done by GPs (κ=0.93, [16]; κ=0.73, [17]).
Regarding ascites, abdominal aortic aneurism and gallstones, the consistency was almost 100% [6]. However it has shown that the ultrasound examination could modify the clinical treatment of abdominal pain in up to 64% of cases, reducing referrals to other specialists up to 40% [18]. In addition, when a Training Program was focused in one single organ/disease (Abdominal Aortic Aneurysm -AAA), the degree of concordance was perfect (κ=1, [19]) and it had high diagnostic validity [20]. The concordance in the diagnosis of profound venous thrombosis between GPs and vascular experts was very good (κ=0.86, [21]); and on hypertensive patients, the concordance degree was 'very high' (κ >0.83, [22]).

The highest discrepancies (GPa, Table 1) were due to the more frequent GPs judgment as 'abnormal', and Radiologist scored as 'normal'. This could be related with the progress of GPs on ultrasound examinations, which generated self-distrust in the initial part and they might mislead some examinations. The difference on academic background and experience was also a variable to take in account, because Radiologists had longer experience and specific training for longer time. So, GPs could not reach the same professional standards only with the practical sessions. However, this would be easy to overcome with time and practical training.
To know the patients perception of ultrasound examination performed by GPs, a survey was designed and anonymously filled out. The new establish approach was successfully accepted in PCC by general public and confirmed by patients' survey that showed high rates of comfort, confidence and care evaluation (Figure 2). In fact, a brief analysis in clinical practice showed that the most prevalent reasons to request an ultrasound examination at the PCC are urological issues (23%), abdominal pain (16%) and hepatic processes (9.4%) (data not shown). It would be very beneficial if GPs could perform, as first screening, ultrasound examinations instead refereeing the patient to the Radiologist in another Hospital. The current study represents a step forward in the quality of ultrasound examinations performed by GPs. Some recommendations has been done by academies and  professionals: structured education with certification, continuous training to keep skills at a sufficient level over time, and choose specific examinations that are more accessible to ensure quality [4, 5, 7, 10, 15]. The sensitivity achieved by GPs appeared suboptimal and ultrasound scan is a reliable tool. Indeed, this would transform GPs ability to achieve a rapid diagnosis for their patients. GPs implemented ultrasound examination after the training program completion; they have been applying this approach on 70% of the general population attended in a PCC. From 2013 to 2016, they performed a total of 10,946 ultrasound examinations. Currently, the new approach has been established in 12 out of 23 PCC in Lleida state.

To our knowledge, this is the first study conducted in Spain that investigated inter-rate agreement and reliability among GPs and Radiologists, after a specified Training Program that covers 1928 patients. Particularly, it developed an ultrasound Training program for GPs, and then analyzed the Program rate of success. In most of the double-checked organs, a high degree of agreement among raters and very high sensitivity of GPs examinations were observed. The research results can be utilized as reliable data to consolidate ultrasound scans in PCC performed by experienced and trained GPs, and to implement actual clinical practices.

Oral consent was obtained from participants throughout a standard medical consultation. Permission was obtained from the two hospitals involved in the study (Hospital Universitari Arnau de Vilanova and Hospital Universitari de Santa Maria). The study was approved by the Institut d'Investigacio en Atencio Primaria (IDIAP) Jordi Gol i Gurina Ethical Committee (registration number p18/077).

"Not applicable", as no personal information is provided.

The datasets used are available from the corresponding author on reasonable request.

The authors declare that they have no competing interests.

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

The authors would like to thank all patients who participated in the Study.

This project was supported by the Territorial Health Management Department of the Institut Catala de la Salut (ICS) in Lleida; the biggest two public Hospitals in Lleida state (Hospital Universitari Arnau de Vilanova -HUAV/Hospital de Santa Maria -HSM); the Departament de Salut de Catalunya (CatSalut); and, the Diputacio de Lleida. The Primary Care Management (DAP Lleida) has been decisive in the development of this program, due to its complexity. Special thanks to Dr. Quinti Foguet, for his useful help and advice with the paper writing and proof reading. We would like to thank the 'Lleida Institut Catala de la Salut Computing Systems Unit' to develop and set up the computer interphase where data was recorded for this study.
Special thanks for two Radiologists, who trained the GPs in the practical part of the Training Program, and to Albert Seoane for proof reading the manuscript. Finally, but not last, we would like to thank the GPs who volunteer and got trained to use the new acquired skills in their jobs to achieve a better clinical practice.

"Lleida ultrasound examination group" is composed by Monica Solanes Cabus, Jesus Pujol Salud, Jesus Alonso Aliaga, Anna Sabate Navarro, Sergi Godia Lopez, Carme Florensa Piro, Mireia Abellana Sangra, Mireia Marin Munoz, Laia Llort Samso, Ramon Mur Garces, Jose Maria Palacin Peruga, Montse Ribes Ribes and Marta Ortega Bravo.

1. Valentin-Lopez B, Blasco-Amaro JA. Plan de uso adecuado de tecnologias de diagnostico por imagen en patologia abdominal en atencion primaria y especializada.: Agencia Lain Entralgo. 2010.
2. Lindgaard K, Riisgaard L. Validation of ultrasound examinations performed by general practitioners'. Scandinavian journal of primary health care.2017;35(3):256-261.
3. Bratland SZ. Ultrasonic diagnosis in general practice. An evaluation study. Tidsskr Nor Laegeforen. 1985 ;105(28):1939-1940.
4. Wordsworth S, Scott A. Ultrasound scanning by general practitioners: is it worthwhile? Journal of public health medicine. 2002; 24(2):88-94.
5. Glaso M, Medias IB, Straand J. Diagnostic ultrasound in general practice. Tidsskrift for den Norske laegeforening. 2007;127(15):1924-1927.
6. Suramo I, Merikanto J, Paivansalo M, Reinikainen H, Rissanen T, Takalo R. General practitioner's skills to perform limited goal-oriented abdominal US examinations after one month of intensive training. European journal of ultrasound. 2002 ;15(3):133-138.
7. Mengel-Jorgensen T, Jensen MB. Variation in the use of point-of-care ultrasound in general practice in various European countries. Results of a survey among experts. The European journal of general practice. 2016 ;22(4):274-277.

8. Fakhri AA. Nuclear medicine consultation: A useful tool in primary care to enable more accurate diagnosis. Journal of family medicine and primary care. 2017;6(2):201-203.
9. Suman A, Schaafsma FG, van de Ven PM, et al: Effectiveness of a multifaceted implementation strategy compared to usual care on low back pain guideline adherence among general practitioners. BMC health services research. 2018 ; 18(1):358.
10. Rikley E, Boillat-Blanco N, Meuwly JY, Breuss E, Senn N: Ultrasonography : an useful diagnostic tool for the general practitioner. Revue medicale suisse. 2017;13(562):990-994.
11. Ministerio de Sanidad SSeI. Cartera de servicios comunes del Sistema Nacional de Salud. In.: Gobierno de Espana. 2012.
12. Catalunya Gd: Pla de salut de Catalunya 2011-2015. In. Edited by Salut Dd. 2012.